MuleSoft Accelerator for Consumer Goods

(0 reviews)

Use case 3 - Trade promotion effectiveness

Visualize your promotions, promotion tactics and account plans in one integrated platform with pre-built assets that integrate Business Intelligence Tools (BI Tools), such as CRM Analytics, into a TPM platform like Salesforce Consumer Goods Cloud.

Overview
Description
Solution definition
Assumptions and constraints
High-level architecture
Sequence diagram
Processing logic
Successful outcome
See also
Downloadable assets

Description

To achieve successful trade promotion effectiveness, it is necessary to have a scalable integration system that can serve as a centralized platform for managing data, facilitating collaboration across different departments, automating manual processes, and offering visibility into the performance of trade promotions. With MuleSoft Accelerator for Consumer Goods, companies can effectively integrate systems to measure the impact of their promotions, make informed decisions on how to optimize them, and respond swiftly to changes in market conditions.

This use case unlocks critical data for analysis and allows key account managers and finance teams to run more efficient and data-driven trade promotions. This solution enables organizations to unify reporting and build visualizations to support their analysis of promotions, promotion tactics and account plans. With access to this information, account teams are better equipped to optimize trade spend ROI and drive sustainable growth.

Ultimately, consumer goods companies and their stakeholders will be able to have a holistic view of their trade spend.

Glossary

Term	Definition
CG	Consumer Goods
CRMA	CRM Analytics
OS	Object Store
RTR	Real Time Reporting
Hyperforce	Salesforce's public cloud architecture.

Goals

Support the extraction, transformation and loading of Promotions, Promotion Tactics and Account Plans from Hyperforce (Salesforce CG Cloud) into Salesforce CRM Analytics for full and delta load.
Orchestrate the Hyperforce APIs to:
- Trigger the export jobs for extracting the CSV files, including Promotion Tactics, Promotion Measures and Account Plans.
- Check the status of the export job for completion and retrieve the files from Hyperforce.
- Commit the Hyperforce APIs, indicating the sucessful processing of the Hyperforce generated files.
Transform the flat files generated by Hyperforce into the dataset format of Salesforce CRM Analytics.
Upload the Promotions, Promotion Tactics and Account Plans datasets to CRM Analytics, which are then ingested into dashboards.
Execute required Recipe API endpoints on Salesforce CRM Analytics to load accurate reports and dashboards.
Ensure Hyperforce system is committed after each successful job completion.

Use case considerations

Hyperforce System APIs enhance the backend Real Time Reporting Gateway APIs to provide a security layer as well as orchestrate the multiple backend RTR Gateway APIs to provide the response in a single API call.
Hyperforce System APIs triggers the export of full and delta load of files based on the incoming request.
Hyperforce System API supports the following:
- Schedule an export job for a given metaData, businessYear and salesOrg.
- Check the status of the job
- Get files related to the job
- Commit the job
A scheduler on the Data Normalization Process API schedules the export of jobs in Hyperforce via the Hyperforce System API.
The Data Normalization Process API checks the status of the job that it scheduled.
Upon the completion of the job in Hyperforce with the status as Ready, the Data Normalization Process API extracts the file from Hyperforce via the Hyperforce System API.
The Data Normalization Process API publishes the files generated for Promotions, Promotion Tactics and Account Plans to an S3 bucket.
AWS Lambda function extracts the .csv.gz file and split the files per million records.
The Data Normalization Process API processes each file in batch mode - reads it from S3, maps the contents of the file according to the Normalization rules and upserts the contents to the Salesforce Einstein Analytics Connector - Mule 4. On successful processing of the file, the Data Normalization Process API moves it to archive folder in S3.
During the processing of first files for every object, KPI Metric information(column names of the file) are appended and stored in S3.
After the ingestion of all the files for the three objects, another scheduler in the Data Normalization Process API picks the KPI Metric file from S3 and uploads it to CRM Analytics.
Once all the split files related to all the three businessObjects or metaDatas are completely ingested, the External Data Recipe API is triggered on CRM Analytics.

Technical considerations

RTR Apex classes should be setup and deployed on the Salesforce CG Cloud.
AWS Lambda function along with S3 buckets and folders should be setup as per the setup guide.
Export configurations should be setup on Salesforce CG Cloud.
Integration User with the necessary permissions should be created for MuleSoft integration with External Data Rest APIs.
- CRM Analytics Plus
- CRM Analytics Plus Admin
- CRM Analytics Platform Admin
CRM Analytics licence should be enabled.
Consumer Goods Cloud Intelligence license should be enabled on Salesforce CG Cloud.

End-to-end scenarios

Promotion Measures created or updated in Hyperforce are reflected in Salesforce CRM Analytics.
Promotion Tactics created or updated in Hyperforce are reflected in Salesforce CRM Analytics.
Account Plans created or updated in Hyperforce are reflected in Salesforce CRM Analytics.
KPI metric dataset including KPIs from Promotion, Promotion Tactic and Account Plan files are reflected in Salesforce CRM Analytics.

Solution definition

The primary set of use cases covered in the current solution involve the synchronisation of Promotion Tactics and Promotion Measures data between Hyperforce and Salesforce CRM Analytics.

Goal

Integrate Promotions, Promotion Tactics and Account Plans data from Hyperforce to Salesforce CRM Analytics.

Main success scenario

A scheduled job in MuleSoft Data Normalization Process API at a configurable frequency will trigger the export API on Hyperforce.
The metadata in the request of the Hyperforce System API will determine the type of objects to be extracted.
Upon receipt of an export job API request within the Hyperforce System API, the RTR Gateway API will return a unique GUID. This GUID can then be utilized to verify the job's completion status.
Hyperforce System API will return the status and the file endpoint to extract the .csv.gz file in response to the successful completion of the export jobs in Hyperforce.
The Data Normalization Process API writes the file to S3.
An AWS Lambda function reads the .csv.gz and extracts the .csv file and splits it per million records.

The Data Normalization Process API

 i.  Processes each file, extracts the column names for generating the KPI metric from the first file.
 ii. Generates the metadata file from the first file.
 iii.Transforms each file as per metadata configuration.
 iv. Uploads it to CRM Analytics via the Salesforce CRM Analytics Connector.

Upon successful completion of all the files:

 i.  Data Normalization Process API extracts the KPI Metric file generated during the first file processing and uploads it to a dataset in Salesforce CRMA Analytics.
 ii.  Data Normalization Process API invokes the REST API for recipes endpoint in Salesforce CRM Analytics.
     a. Invokes the CRM Analytics Rest API (/services/data/v56.0/wave/folders) to get the result of all apps present in CRM Analytics.
     b. Checks the “templateSourceId” parameter for a value of “sfdc_internal__CG_TPE_Analytics”.
     c. If it exists, checks the “applicationStatus” parameter for “completedstatus”. If it doesn't exist, the Data Normalization Process continues with step 12.
     d. Extracts the “name” of the apps and stores these relevant names.
     e. Invokes to recipes endpoint (/services/data/v57.0/wave/recipes) and searches for recipes which starts with <name>_CG_TPE_Workflow.
     f. Validates the response to see if the name matches with dataSetName configured in the property file.
         i. For Promotion dataset: $.recipeDefinition.nodes.LOAD_DATASET0.parameters.dataset.name
         ii. For Promotion Tactic dataset: $.recipeDefinition.nodes.LOAD_DATASET1.parameters.dataset.name
         iii. For Account Plan dataset: $.recipeDefinition.nodes.LOAD_DATASET17.parameters.dataset.name
         iv. For KPI List dataset: $.recipeDefinition.nodes.LOAD_DATASET14.parameters.dataset.name
     g. If each dataSetName matches, extracts the recipeId and targetdataflowId.
     h. Triggers the recipes based on recipeId and targetdataflowId.

The Data Normalization Process invokes the Hyperforce System API to commit the job in Hyperforce, indicating the completion of the successful processing of the files related to the job.

Assumptions and constraints

The following is used to guide or constrain the solution design at a high level:

CRM Analytics Limits.
Maximum file size per external data uploads: 40 GB
Maximum file size for all external data uploads in a rolling 24-hour period: 50 GB
Maximum number of external data jobs per dataset that can be run in a rolling 24-hour period: 50
Maximum number of characters in a field: 32,000
Maximum number of fields in a record: 5,000 (including up to 1,000 date fields)
Maximum number of characters for all fields in a record: 400,000
Salesforce Einstein Connector requires the metadata of the file. This is provided with the mule implementation project.
The solution currently supports one salesOrg per businessObject.
No spaces are allowed in the dataSetName in Salesforce CRM Analytics.

High-level architecture

Sequence diagram

The diagram below illustrates the sequence of processing the Promotion Measures and Promotion Tactics to Salesforce CRM Analytics from Hyperforce.

Processing logic

The primary handling and orchestration of Promotion Tactics and Promotion Measures will be implemented in the Data Normalization Process API. This process can be described as follows:

A scheduler is set up for each businessObject or metaData in the Data Normalization Process API.
The Data Normalization Process API checks the Object Store to determine if any previous job with the same configuration of metaName, businessYear and salesOrg is still in "InProgress" status.
When an entry is found, no new job is scheduled. In the absence of an entry, the following steps are performed:
The Data Normalization Process API invokes the Hyperforce System APIs to initiate the export jobs in Hyperforce to retrieve Promotions, Promotion Tactics and Account Plans based on the configured business year and sales org.
Hyperforce System API will invoke the corresponding backend RTR Gateway APIs to trigger the export job.
The Data Normalization Process API checks the status of the job for Ready status in the Hyperforce System API at a configured frequency of 30 minutes.
Once the status of the job is Ready, the Data Normalization Process API invokes the Hyperforce System API to get the file in .csv.gz format.
The Data Normalization Process API writes the file to S3 in the inbox folder.
A Lambda function setup in AWS reads the file from the inbox folder, extracts the .csv and splits it per million records and writes it back to the inbox folder.
The Data Normalization Process API does the following:a. Reads the split files from S3 and processes them.b. For every first file, extracts the column names and appends them to KPI Metric file.b. Transforms the file to CRM Analytics format.c. Uploads it to CRM Analytics using the Salesforce CRM Analytics Connector.
After all the split files related to all the three businessObjects or metaDatas are processed, the Data Normalization Process API reads the KPI Metric file from S3 and uploads it to a dataset in CRM Analytics.
The Data Normalization Process API invokes the recipe APIs on CRM Analytics.a. Invokes the CRM Analytics Rest API (/services/data/v56.0/wave/folders) to get the result of all apps present in CRM Analytics.b. Checks the “templateSourceId” parameter for a value of “sfdc_internal__CG_TPE_Analytics”.c. If it exists, checks the “applicationStatus” parameter for “completedstatus”. If it doesnt exist, the Data Normalization Process continues with Step 12.d. Extracts the “name” of the apps and store these relevant names.e. Invokes to recipes endpoint (/services/data/v57.0/wave/recipes) and searches for recipes which starts with _CG_TPE_Workflow.f. Validates the response to check the name matches with dataSetName configured in the property file.
```
i. For Promotion dataset: $.recipeDefinition.nodes.LOAD_DATASET0.parameters.dataset.name.
ii. For Promotion Tactic dataset: $.recipeDefinition.nodes.LOAD_DATASET1.parameters.dataset.name.
iii. For Account Plan dataset: $.recipeDefinition.nodes.LOAD_DATASET17.parameters.dataset.name.
iv. For KPI List dataset: $.recipeDefinition.nodes.LOAD_DATASET14.parameters.dataset.name.
```
g. If all the four dataSetNames match, extracts the recipeId and targetdataflowId.h. Triggers the recipes based on recipeId and targetdataflowId.
The Data Normalization Process API invokes the commit job on the Hyperforce System API.
The Hyperforce System API sends a request to the backend RTR Gateway API to commit the job in Hyperforce.
Until a job is committed in Hyperforce, the files related to that job are considered full load. Once a job is committed in Hyperforce, the files related to the job are considered delta.

Successful outcome

After completing the update processing for all target systems successfully, the following conditions will be met:

Promotion Measures from a particular year in the Salesforce Consumer Goods Cloud are synchronized with Salesforce CRM Analytics.
Promotion Tactics from a particular year in the Salesforce Consumer Goods Cloud are synchronized with Salesforce CRM Analytics.
Account Plans from a particular year in the Salesforce Consumer Goods Cloud are synchronized with Salesforce CRM Analytics.

Systems involved

Hyperforce
CRM Analytics

Setup instructions

Before you begin

The Getting Started with MuleSoft Accelerators guide provides general information on getting started with the accelerator components. This includes instructions on setting up your local workstation for configuring and deploying the applications.

Downloadable assets

Type	Custom
Organization	Mulesoft Inc.
Published by	MuleSoft Solutions
Published on	Jun 20, 2023